runtime: recoverable hardware faults with fault-site stacks (dynamic libunwind + FP chain)#2028
Open
cpunion wants to merge 4 commits into
Open
runtime: recoverable hardware faults with fault-site stacks (dynamic libunwind + FP chain)#2028cpunion wants to merge 4 commits into
cpunion wants to merge 4 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
287539a to
3351dfa
Compare
…libunwind + FP chain) Hardware faults — SIGSEGV/SIGBUS and previously-fatal SIGFPE — now convert to ordinary recoverable Go panics, and the unrecovered traceback shows the fault-site chain: C frames down through the Go callers. - A SA_SIGINFO handler captures the interrupted context; the handler's own frame-pointer chain dead-ends at the signal trampoline, which is why the ucontext pc/fp is required. - The capture prefers a dynamically-resolved libunwind (dlopen/dlsym at install time only — no link-time -lunwind; LLGO_DYNUNWIND=0 disables): DWARF/compact-unwind stepping survives C frames compiled without frame pointers, and the nongnu flavor's unw_get_proc_name reads .symtab, naming static C symbols dladdr cannot see (they otherwise display under a neighboring Go function via nearest-below). Where unwind info runs out, the walk resumes along the FP chain from libunwind's final cursor. Flavors: darwin libSystem, linux nongnu (arch-prefixed symbols), linux LLVM (context translated). - Only man-page async-signal-safe unw_* calls run in the handler (resolution and a lazy-state warm-up happen at install). Fault-context walks probe page readability (msync) before dereferencing — an arithmetic-valid frame pointer can still point into an unmapped hole, and faulting inside the fault path would recurse; a re-entered handler restores the default disposition for one clean core. - The unrecovered dump goes through a new PanicTraceback hook (gc-style frames via the funcinfo tables; libunwind's name for dot-less C symbols); non-fault panics keep the existing clite dump. Verified: darwin/arm64 (libSystem flavor); linux/amd64 with a -fomit-frame-pointer C chain — the FP-only walk recovers 2 frames with a misattributed name, the dynamic path recovers the full chain with correct static names. Overlaps with the fault half of xgo-dev#2026 (panic-site snapshots); whichever lands second rebases to drop its duplicate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Preserve errno across fault-handler installation: the dlopen probes for libunwind flavors leave errno=ENOENT on systems without libunwind, and the leaked value surfaced as a spurious non-nil err on the first two-result cgo call (broke _demo/go/cgo on CI ubuntu). Also preserve errno in llgo_mem_readable (msync probe). - Prime the page size at handler install instead of lazily calling sysconf inside the signal handler. - Capture frame symbol names in the handler only where the resolver is in-memory (darwin libSystem). nongnu's unw_get_proc_name reads /proc/self/maps and mmaps ELF .symtab — not async-signal-safe — so it is now opt-in via LLGO_DYNUNWIND_NAMES=1; frame pcs and the resume fp are still captured. - Move errno/mman/unistd includes and the llgo_pagesz declaration above their first use.
C frames have no line table entry; print 'pc=0x<addr>' (offline-resolvable via addr2line/atos) instead of the '???:0' placeholder. Stop the walk at the first out-of-text frame once the module's own frames have printed: everything below main (__libc_start_main, _start) is startup plumbing. Leading out-of-text frames still print, so a fault inside a named libc/ C-library function keeps its context.
3351dfa to
2e095ae
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based directly on #2019 (contains #2012/#2016): hardware faults become recoverable Go panics, and the unrecovered traceback shows the fault-site chain — C frames down through the Go callers — captured from the signal ucontext with a dynamically-resolved libunwind, falling back to the frame-pointer chain.
Productizes the
codex/exp-dyn-unwindexperiment (feasibility validated on darwin/arm64 + linux/amd64 + linux/arm64).What changes
SA_NODEFERbecause recovery longjmps out through savemask=0 jmpbufs: an auto-blocked signal would stay blocked and turn the next genuine fault into a forced core._wrap/dynunwind.c): resolved via dlopen/dlsym at install time only —readelf -dstays free of libunwind;LLGO_DYNUNWIND=0forces the FP walk. Three flavors: darwin libSystem (in-memory name resolution, on by default), linux nongnu (_ULx86_64_*-prefixed; itsunw_get_proc_namereads.symtaband can name static C symbols, but does so via/proc/self/maps+ ELF mmap — not async-signal-safe — so name capture is opt-in viaLLGO_DYNUNWIND_NAMES=1; frame pcs and the resume fp are always captured), linux LLVM (context translated; frames only). Where unwind info runs out, the walk resumes along the FP chain from libunwind's final cursor.msyncreadability probe. Fault-context walks probe page readability before dereferencing — an arithmetic-valid frame pointer can still point into an unmapped hole, and faulting inside the fault path would recurse; a re-entered handler restores the default disposition for one clean core. The errno preservation is load-bearing: the nongnu warm-up leaveserrno=ENOENT(vdso has no on-disk file), which surfaced as a spurious non-nilerron the first two-result cgo call (v, err := C.f()) and broke_demo/go/cgoon any system with libunwind installed.PanicTracebackhook prints gc-style frames for fault panics (funcinfo tables for Go frames, libunwind names for dot-less C symbols); non-fault panics keep the existing clite dump.Decisive result (linux/amd64, C helpers built
-fomit-frame-pointer,LLGO_DYNUNWIND_NAMES=1)FP-only walk: 2 frames, fault pc misattributed to a neighboring Go symbol. Dynamic path — verbatim output of the nofp probe (chain:
fx.main -> viaGoNofp -> cexc_nofp_segv -> cexc_nofp_mid x3 -> cexc_nofp_leaf, fault inleaf):C frames carry no line-table entry, so the indent line prints the raw pc (offline-resolvable with addr2line/atos) instead of a
???:0placeholder, and the walk stops after the module's frames — the startup tail (main,__libc_start_main,_start) is plumbing gc tracebacks never show either. A fault inside a named C-library function still prints its leading out-of-text frames.Reading the frames:
cexc_nofp_leafis inlined intocexc_nofp_midby the C compiler, so the fault pc reports asmid+0x1e; the next twomidframes are the recursion's return addresses;viaGoNofpis inlined intofx.mainby llgo, whose frame carries the exact Go file:line. These are non-exported C symbols — invisible to dladdr — named via nongnu's.symtabreader. With names off (the linux default) the frame count, order and Go frames are identical, but the C pcs fall back to the table's nearest-below attribution and print a confidently wrong Go symbol (verbatim):This is the pre-documented C-in-Go-text-range hole; the fix (hole sentinels so foreign pcs print as unknown/named instead of a neighboring Go function) is the P4 follow-up in #2004. Until then,
LLGO_DYNUNWIND_NAMES=1gives correct C names on the nongnu flavor at the cost of async-signal-unsafe work in a dying process's handler.Known limitation
The handler is installed from the runtime shim's init (
lib/runtime); a program that imports nothing which links the publicruntimepackage keeps the pre-existing behavior for C faults (raw core dump — same as before this PR, so no regression, but such programs do not gain fault recovery either). Moving the install into the runtime core is possible if the startup dlopen probes are acceptable for every binary.Relationship to the other stage-5 PRs
Overlaps with the fault half of #2026 (which also carries Go-panic snapshots and recovered-
debug.Stacksplicing on the #2023 base); whichever lands second rebases to drop its duplicated fault machinery. Naming normalization (main.prefix) and pcline conformance live in #2023 and apply on top once merged.Validation
test/gogreen; probes verified with the libSystem flavor engaged (LLGO_DYNUNWIND_DEBUG=1)._demo/godemos, fault regressions, statement suites green.-fomit-frame-pointercomparison above._demo/go/cgoerrno regression counterfactually verified on linux/amd64 with nongnu libunwind installed: fails on the previous head, passes with the errno preservation; also passes with libunwind absent.TestCFaultRecoverable(three sequential faults — exercises SA_NODEFER),TestCFaultTraceback(C frame-name assert darwin-only).🤖 Generated with Claude Code